Decision analysis methods guide
Agricultural policy for nutrition
Eike Luedeling\(^1,^2\)
Cory Whitney\(^1\)
\(^1\)University of Bonn, Germany; \(^2\)World Agroforestry Centre, Kenya
List of abbreviations
Summary
It is often very difficult to make accurate projections about how interventions will affect the real world and to thereby develop effective implementation plans, monitor progress and evaluate project impacts. This is due to a variety of factors including lack of data, complex impact pathways, and risks and uncertainties that are difficult to factor into intervention planning. Scientific approaches to produce reliable impact projections are rarely applied in agricultural development, but Decision Analysis techniques commonly used in other fields have the potential to improve development decisions. This manual outlines a Decision Analysis approach that can help decision-makers efficiently allocate resources for the most effective policy impacts.
The procedures outlined in this manual feature the construction of causal models – models that describe the mechanisms through which intervention impacts will be delivered – that are co-developed by experts, stakeholders and analysts through facilitated participatory processes. These models are then formalized as Bayesian Network (BN) models, a modeling approach that has been widely applied in a range of disciplines, including medical sciences, genetics, environmental sciences, and legal reasoning. BNs allow for the formal representation of causal models, such as intervention impact pathways. They can work effectively with incomplete information, combine expert knowledge with other sources of information, and they allow adequate consideration of risk.
This manual illustrates the use of participatory workshops that convene experts on the systems, stakeholders involved in ongoing or prospective projects, and analysts. These teams can jointly develop impact pathways for the interventions, which can be formalized into quantitative BN models. After several rounds of feedback elicitation, and inclusion of data from experts and other sources, stochastic simulations can be run to determine the likely impacts of the interventions. Results can be presented back to stakeholders for feedback.
Through the tools laid out in this manual critical uncertainties in the models of intervention impact pathways can be identified. These high-value variables can determine uncertainty about project outcomes. Further measurement or dissagregation of these variables could greatly support decision-making processes.
By demonstrating improved intervention decisions with minimal additional investment and improved tools for intervention decision modeling, we hope that this approach will be widely adopted and used to enhance the efficacy of development activities.
Introduction
The development community faces increasing demand to credibly link research and development activities with progress towards the envisioned outcomes (Shepherd, Luedeling, & Whitney, in preparation). Better planning tools for interventions that target complex systems are urgently needed, especially in developing countries where uncertainties are large. However, especially where quantitative impact predictions are requested, stakeholders are often left guessing about development outcomes, because they lack reliable tools to forecast impacts. Methodologies that address these uncertainties could transform the way development is done and greatly enhance the efficacy of its activities. Such methods could stimulate thorough scrutiny of research priorities and direct resources to where they lead to the greatest impacts (Luedeling & Shepherd, 2016). For this positive effect to materialize, however, the development community needs better approaches for planning for impact.
One of the central difficulties in planning for impact is uncertainty. It is rarely possible to precisely predict the impacts of agriculture for nutrition interventions because critical factors, like adoption rates, yield increases, the performance of a particular tree or crop in a new environment and future weather cannot be forecast precisely. Additionally, many development interventions are implemented in risky environments, where extreme weather events, conflict, poor anticipation of cultural preferences, political interference or other risk factors can dramatically disrupt progress at any time (Luedeling et al., 2015). It is worth noting, however, that for research that is actually worth doing, results cannot be forecast with certainty (otherwise the research would not be necessary).
Making impact projections in this environment, especially where precise numbers are expected, is very difficult, researchers and development workers often find themselves in an ethical quagmire, torn between the perceived need to honestly evaluate risks and the temptation to let wishful thinking guide their estimates. The latter may lead to overly optimistic assumptions and high impact projections that may raise the chance of political and donor support but are essentially unrealistic. This conflict of interest is not only a problem for project proponents – it also compromises the ability of stakeholders to compare impact ‘promises’ across proposals or reported impacts by different projects.
What is currently lacking is a set of reliable methods to produce robust impact projections that take into account the host of uncertainties and risks that research and development activities are faced with. Such methods should be based on quantitative representations of impact pathways that capture the causal mechanisms of impact delivery. This manual aims to provide documentation for disseminating an approach based on Bayesian Networks (BNs) for impact pathway modeling. It seeks to establish Bayesian Networks (BNs) as a widely used analysis tool for development decisions related to agriculture for nutrition.
Predicting impacts of Agriculture for Nutrition activities
Many activities in agricultural research and development aim at improving nutrition, but they are often unable to articulate clearly how nutrition objectives will be achieved and to what degree. Agricultural systems in developing countries are complex, and few agricultural interventions can be expected to impact such systems in a linear way. Thus there is a need for new approaches for analyzing the impacts of agricultural interventions on food and nutrition systems.
The success of an intervention will always depend on a number of factors that interact in ways that would be difficult or impossible to predict with precision. Some examples of these difficult to measure factors are the so-called ‘intangible’ factors such as people’s perceptions of healthy food and their food preferences. The nutritional status of a country’s population is determined by many such factors, including, of course, the nutritional value of the food people eat, but also a complex interplay between the food environment, household economics, health, education, and agricultural value chains (Waage, Hawkes, & Turner, 2012). Thus, many pathways may have the potential to improve national nutrition, e.g. through higher nutrient contents in crops (DellaPenna, 1999; Nestel, Bouis, Meenakshi, & Pfeiffer, 2006), greater nutritional diversity (Hoddinott & Yohannes, 2002) or improved awareness about childhood nutrition (Ruel, Alderman, & Maternal, 2013). For any given context, however, it is difficult to decide a priori, which pathway will be most effective. Some pathways may not produce positive outcomes at all, if, for instance, the value chain degrades the nutritional value of the food, or if certain foods never reach vulnerable groups (e.g. children or lactating women).
Credible impact pathways regarding agriculture for nutrition should reflect their complexity. However, there is currently a severe shortage of practical methods that allow credible analysis of these complex systems. Most conventional scientific approaches are unable to deal with this complexity. Opportunities for controlled trials are very limited, especially at low cost, and simple statistical tools (regressions, correlations) do not provide much information about the way that drivers of agricultural systems are related to nutritional (and other) outcomes.
Use and analysis of impact pathways have helped to show how interventions function, where they are lacking and what can be done to improve them. Leroy, Ruel, & Verhofstadt (2009), for example, used impact pathway models to review the effect of cash transfer programs on child nutrition outcomes. Olney, Talukder, Iannotti, Ruel, & Quinn (2009), used impact pathway models to evaluate the maternal and child health and nutrition effects of a homestead food production program in Cambodia and found that household-level benefits from the program did not translate into significant improvements in maternal and child health and nutrition. Both studies found a major gap in implementer and stakeholder knowledge about how the programs improve nutrition and identified this as a major obstacle to the interventions.
This manual provides a step by step guide to help synthesize expert knowledge and other sources of information into BN models that provide credible probabilistic projections of the impact of decisions. The methods described can be applied to multiple nutrition outcomes. The resulting models, as well as the participatory process from which they emerge, can be used to define useful metrics for monitoring progress towards nutrition outcomes.
… a Bayesian approach, which seeks to express the current state of uncertainty on everything that matters to a decision, can help to focus the measurement effort on areas that can narrow uncertainty to reduce ambiguity in the decision. Analysts can then update BNs based on the new information. Whether a factor is seemingly easy or difficult to measure or has existing data available has no bearing on its inclusion. Omitting an important factor is essentially prescribing it as valueless.
This is then complemented with innovative group work techniques for eliciting expert knowledge to construct a logical framework to describe system interactions and outcomes (i.e. an impact pathway). Expert knowledge is thereby used to generate BN model structures (Bolger & Rowe, 2015; Kuhnert, Martin, & Griffiths, 2010; Papakosta, Xanthopoulos, & Straub, 2017) and these are then integrated into model calculations (cf. Yet et al., 2016).
Decision Analysis
Decision Analysis provides a framework for development research, the objective of which is to make better decisions by gaining insights into what actions could most increase multiple benefits given stakeholder preferences while minimizing costs and risks. Abbas & Howard (2015) illustrate how the foundations of Decision Analysis provide the norms for decision making. The basic Decision Analysis approach seeks to increase benefits and decrease risks on a continuous basis through the intervention planning and implementation process. The insights gained through the Decision Analysis approach include better understanding of the magnitude of the trade-offs, among different development objectives relative to the preferences of different stakeholder groups. The ultimate aim is to lead to clarity of action for decision makers.
The basic steps in the Decision Analysis process, see , address the questions of both why and how decisions are formulated and factors are measured. Hubbard_2014 outlines some of these approaches as part of his Applied Information Economics (AIE), which he calls the ‘Universal Modeling Approach’, since AIE has the basic premise that if something has an affect, it must be observable, and if it is observable it must be measurable. Decision Analysis and AIE are a collection of decision theory and risk analysis tools which use:
shows a technical diagram of the Decision Analysis process adapted from Shepherd et al. (in preparation). The process outlined is followed in cooperation with key stakeholders and experts to improve the design of policy and interventions and monitor their impacts. The loop in the top half of the diagram describes the process that evaluates different alternatives in relation to the decision goals, whereas the lower loop uses value of information analysis to determine what should be measured to clarify the decision. There are iterative feedback loops throughout the process.
Summary of the decision modeling proces; the sequence of activities in the decision modeling approach (adapted from Shepherd et al. in preperation).
The merits of the application of the Decision Analysis tools for development decisions have been further described in detail. Luedeling & Shepherd (2016) point out that Decision Analysis solves the problem of data gaps, which has often prevented research from comprehensively and holistically forecasting decision impacts. It also allows explicit consideration of risks and variability.
Decision Analysis tools allow for decision making that draws on all appropriate sources of evidence rather than rule out intangible and hard to measure aspects of a decision. Luedeling & Shepherd (2016) point out that, in Decision Analysis, a model should include all the factors and all the important decision impacts that experts consider relevant, regardless of data availability. Shepherd et al. (2015) points out that Decision Analysis tools are particularly useful in development contexts, where data are often sparse. One very useful aspect of the approach is that expert knowledge can be used to fill in the knowledge gaps and avoid missing important factors when deciding about development interventions.
Bayesian Networks
Bayesian Networks (BNs) are a powerful tool for analyzing causal relationships in complex systems, for integrating different sources of knowledge, and for dealing effectively with imperfect information. BNs constructed with participation from key stakeholder groups could provide a rapid robust way of applying existing knowledge to forecast intervention outcomes. This technique is practically absent from the planning of agricultural interventions for nutrition outcomes, even though it is well suited for many of the analytic challenges that researchers in this field have been struggling with. Introduction of BNs into this field could greatly enhance the way projects are planned, monitored and evaluated.
BNs have been in use since the 18th century to determine the probability of an event based on prior knowledge about the event. According to Nielsen & Jensen (2009) BNs are directed acyclic graphs consisting of nodes arcs and probability tables underlying the nodal relationships. This means that BNs are based on direct causal relationships and do not contain logical loops, instead they are formulated as directed and acyclic models, as in a family tree. A BN is a network of probabilistic relationships between variables (called nodes in the Bayesian nomenclature), referred to as child and parent nodes according to their arrangement in the model (cf. ). Conditional Probability Tables (CPTs) are the core elements of BN models (Fenton & Neil, 2012). They are used to define the probabilities for each state of each child node conditional on its parents (Papakosta et al., 2017). This allows analysts and experts to represent qualitative nodes as abstractions of the underlying continuous quantities. As an example in the variable Food Preferences was considered important for experts, but was difficult to quantify. Therefore it is represented as having the states ‘very low’, ‘low’, ‘medium’, ‘high’, and ‘very high’. The CPT for this variable is populated with the probabilities for these different states given the states of the parents Location and Nutrition awareness.
BN models are distinct from other forms of statistical models, in that they focus on determining an optimal graphical model to describe probabilistic inter-relationships among processes rather than on specific measurement data. BN models are a multivariate technique, which can accommodate one or many dependent variables. The approach can be used to investigate risk factors and causal pathways, which is important in health-related systems (cf. Lewis & McCormick, 2012). The key distinction between standard multivariate analyses and BN models is that multivariate regressions seek to identify covariates associated with some outcome variables. BN models, on the other hand, attempt not only to identify associated variables but also to separate them into those that have a direct and those that have an indirect influence on the outcome variables (Lewis, 2012). This gives BN models the potential to reveal far more about key features of complex systems (Constantinou, Yet, Fenton, Neil, & Marsh, 2016; Lewis & McCormick, 2012) and may make them preferable to standard approaches for inferring statistical dependencies from complex observational data (Korb & Nicholson, 2004; Lewis & McCormick, 2012). According to studies such as Papakosta et al. (2017), another major advantage of BN models is that they facilitate integration of information from various sources into a single model. Yet et al. (2016) also shows how BNs can be used to build predictive models of impact pathways that incorporate both hard data and expert judgment. Despite their widespread applications BNs have not been exploited in the agriculture/nutrition domain. Yet, BNs are highly suited to this domain. For example, Henderson & Burn (2004) show how BNs are able to combine information from various sources, such as hard data and expert knowledge, into comprehensive causal models that result in more accurate impact forecasts than using data or expert knowledge alone.
This manual shows how to adapt and apply BN-based procedures for developing comprehensive impact pathways for agriculture for nutrition interventions and demonstrate their ability to aid in decision-making under multiple uncertainties and imperfect information. The manual acts as a guide to construct robust and reliable impact models by convening experts in the fields of agriculture and nutrition, as well as related disciplines. The modeling approach has several advantages. First, it allows for comparison of the prospects of different interventions for improving nutrition security, thus helping to identify the most promising approaches. Second, it allows for the consideration of risks to intervention success and identification of weak links in the impact pathway that require particular attention by intervention planners. Third, it permits the inclusion of difficult to measures factors that are commonly omitted from models, opening opportunities for holistic research. Lastly, it helps prioritize metrics that will be critical to monitor during implementation – especially those variables that have large residual uncertainty and a large potential impact on intervention outcomes.
Steps in the Decision Analysis process
Decision framing
Decision framing is the first part of the decision analysis process, it determines the boundaries of a decision and is the most important aspect of making a good decision. Decision framing starts by identifying both a decision to be modeled and the relevant experts. This is not as simple as it sounds and should be undertaken carefully and with a lot of forethought. Consideration of the criteria which define a decision can help to help add structure to this important step in the process:
As Shepherd et al. (in preparation) points out, in order to to achieve clarity in decision making each of the above elements, and the preferences of the decision maker, need to be carefully defined.
Once the decision has been identified, the decision analysts then identify and convene relevant decision-makers, stakeholders, and any additional experts. We find that around 20 experts makes for a manageable workshop, more can be cumbersome, especially when dealing with model building in plenary.
It is important to have a number of different types of knowledge holders involved in the development of the impact model. Therefore, expert selection should seek to gather experts with knowledge on as many aspects of the decision as possible. As Fenton & Neil (2012) points out, expert knowledge is often vital in identifying critical, underlying causal factors that affect risks and opportunities that would otherwise have been missed based on available data or statistical models. The experts selected for modeling agriculture for nutrition decisions should represent a mix of knowledge holders such as academic institutions (e.g. nutritionists and agronomists), government institutions, local villages and development organizations (see Luedeling, 2017).
Generating a graphical model
The next step in the Decision Analysis process is to jointly develop a decision model (). In the workshop setting it is important for the analyst to gather all the variables that the expert group agrees are logically important to describe the impact pathway of the decision and include them in the model. This should be done regardless of the ease of measurement of the individual variables.
Before engaging in this process the overall context for the model (the decision that was identified) should be defined explicitly and agreed upon in plenary discussions with the identified experts. Once this has been done, building the graphical model can begin. In our approach we start from the decision framing step through to the model development by asking experts to work together and peer review each others work.
Bolger & Rowe (2015) and Bolger & Wright (2017) point out the various challenges in handling expert knowledge, which include both poor judgment regarding probability and high levels of variation among experts. These problems occur because experts are not trained to formulate reliable representations of uncertainty, many lack experience with probability and few learn to express their uncertainty as probability distributions. Here we outline several important steps that should be taken to aid in the process of collaboratively building model structures and making variable estimates. These will help to ensure accuracy in the process.
Colaborative approaches
Tools are available to help overcome these problems. According to Bolger & Rowe (2015) it is possible to create better conditions for gathering expert knowledge. These include offering experts experiences in making estimates for well-defined targets, offering tools on which to base their estimates, and offering regular and usable feedback about the accuracy of their estimates. Bolger & Wright (2017) experts can be a source of quality data about the future. Ensuring the quality of this data starts by selecting the best experts, training experts in the normative aspects of anticipation and combining judgments from several experts.
Here we illustrate some useful approaches for combining experts probability distributions, using the accumulated information from multiple experts. In this approach a workshop is held and graphical models are developed by individual experts and then peer reviewed by other experts. These approaches allow analysts to obtain as much information as possible and gather data that represent a summary of expert opinion. More examples of these tools in practice are shown by Clemen & Winkler (1999) and Bolger & Wright (2017).
has been adapted from Whitney et al. (in preparation), it illustrates a process that can be used for eliciting graphical representations of decisions from expert groups to be used in developing a BN. This approach is generally outlined in the work of Iqbal & MacLean (2010), who consulted experts in repeated group meetings to build a BN for defoliation prediction by sawfly infestations in Newfoundland. Each expert was asked to create a BN model and this was then peer reviewed by other experts. In our approach the process begins with breaking the decision down into several important questions in plenary discussions. Random interchanging working groups of experts are then led through three stages of collaborative thinking for each question that is brought up, these are:
The steps defined above and shown in are designed to help experts interact, brainstorm and help reach common understanding about impact pathways. Through this approach, experts can explore details of the expected impacts, disaggregate the impact pathway into the intermediate steps and identify all the influencing factors that they consider important to the decision (i.e. draw a model of nodes and edges; cf. ).
Process used for eliciting graphical representations of decisions from expert groups to be used in developing a BN
The steps shown in are repeated until all experts have worked on each question and are satisfied that all specific relationships have been identified. This iterative process of building a model in the workshop is illustrated conceptually in . Participants should be encouraged to discuss any factors they deem important for the decision, in particular the various costs, benefits, and risks associated with interventions, as well as the objectives and concerns of decision-makers and stakeholders. The model should have the broad aim to describe the effects of agricultural decisions on specific nutrition outputs such as hunger (a.k.a. global energy and macronutrient deficiency).
Following this approach, it is important that the analysts have an overview of the process with the objective of building a final working BN model in mind. Analysts should work to guide the experts so that they think about how model parameters interact in a logical way. For example the work of Whitney et al. (in preparation) illustrates the use of the EKE approach to build an impact model which aims to deliver the probabilities for different states of malnutrition under a policy decision and to relate this directly to a monetary value for calculating variables of importance (see )
Resulting models can then be brought before the whole group of experts for plenary discussion and re-drawn, aiming for common understanding about the relationships in each model. The end result should be one model per question with the contributions of all experts. These can then be combined into one large impact pathway model. Corrections and further feedback can be gathered for model verification as a final stage of model development.
Calibration training
The next step of the modeling procedure follows procedures, outlined by Hubbard (2014), known as ‘calibration training’. Experts are trained, using well-established ‘calibration’ procedures to estimate their state of uncertainty and thereby increase their capacity to provide accurate estimates by reducing errors of judgment. Through the process, experts learn to improve their ability to estimate their own state of uncertainty and thereby reduce errors of judgment, e.g. under-confidence or overconfidence (i.e. give correct estimates 90% of the time when they have 90% confidence, Hubbard, 2014). As both Hubbard (2014) and O’Hagan et al. (2006) show, through this training the experts learn to minimize potential biases in probability estimation.
Calibration training consists of several exercises aiming to reveal to the participants their personal biases (overconfidence or underconfidence) by assessing their performance on a set of trivia questions. Through these exercises experts are trained to assess their subjective uncertainty and express it as a Confidence Interval (CI). This interval has a predefined chance (e.g. 90%) of containing the right value. Perfectly calibrated people should get around 90% of answers correct, and any deviation (outside a narrow band of stochastic variation) from this optimal figure indicates estimation bias. Experts are calibrated through repetition and feedback on the exercises. The training is divided into several stages:
The examples given by Hubbard (2014) and Luedeling et al. (2015) show that through the use of these procedures, experts learn to give estimates that are neither too vague (under-confidence) or too specific (overconfidence). The procedures seek to help participants become accurate estimators. Accuracy, in this context, does not mean precision but rather refers to the ability to produce an accurate range of probable estimates of different possible states of the variables of interest. Therefore, the procedures outlined in this appraoch seek to provide experts with the skills necessary to represent uncertainty explicitly as probabilities of different possible states of the world. Estimations of the nodes states within the BN can begin once the experts have been through the training and the analysts are satisfied that the experts are adequately calibrated. More examples of these calibration procedures and instructions on their application is provided in detail by Hubbard (2014).
Model quantification
For final model quantification the analyst converts the conceptual model into a mathematical model, translating stakeholder inputs into equations as accurately as possible. Following Luedeling et al. (2015), all estimates can then be consolidated into one single distribution for each model parameter.
As mentioned in BNs are based on simplified node structures, called Conditional Probability Tables (CPTs) specifying the relationships between parent and child nodes. CPTs can sometimes contain large numbers of conditional probabilities, especially where several parents with multiple possible states are involved. Estimating large numbers of probabilities can be overwhelming for experts. For this reason, Fenton, Neil, & Caballero (2007) warn that the overwhelming nature of this CPT filling process can lead to inconsistencies. The works of both Cain (2001) and Marcot, Steventon, Sutherland, & McCann (2006) also warn us that these inconsistencies have the risk of leading to unreliable models. Luckily, there are several tools available to simplify the process of filling CPTs. For example Fenton et al. (2007) found that experts find it easier to understand and express relationships through the use of weighted averages and ranked nodes. Rather than fill CPTs directly, experts can be asked to provide a typical distribution of each variable (prior probability) together with the strength of the response and a weighting factor for each state of the child node and each state of the parent nodes. In we provide an example of an input sheet that can be used for eliciting expert knowledge from four main aspects of the variable of interest:
gives an example of a tool for translating expert knowledge into a Conditional Probability table (CPT) for use in a Bayesian Network (adapted from Whitney et al., in preparation).
This approach helped us to gather expert knowledge for all model parameters. These can then be used to calculate the CPTs (see Luedeling, 2017) using the likelihood method as described by Hansson & Sjökvist (2013). As described by Kemp-Benedict, Bharwani, Rosa, Krittasudthacheewa, & Matin (2009) experts can provide influence weights rather than whole tables of probabilities.
Once completed and verified with the literature and other sources, the BN can be shared with experts again for verifying its logical consistency and receiving final feedback.
Software resources for direct application of these calculations are given later in the manual. Procedures for producing CPTs from these inputs can be implemented in the decisionSupport package for R (2017). This is outlined in .
Software for modeling Bayesian Networks can be used to build interactive graphical models and for calculating the BN. In we outline some of the tools available in the AgenaRisk package (Fenton & Neil, 2016), useful for calculating the BN model.
Example Bayesian Network
An example BN model of a policy strategy in Uganda is shown in . The model includes 29 nodes to describe the impact pathway from the strategy decision (Vision 2040; left) to household nutrition (hunger and micronutrient deficiency; right). The model includes two policy related nodes (nutritional awareness and promotion of exports), five economic nodes (land tenure, location [proxy for access to goods and services], exported production, occupation, and food access), seven social variables (displacement of farmers, nutritional education and sensitization, traditional knowledge, nutrition awareness, diversity of agricultural systems, food preferences, and nutritional quality), and seven nodes related to both micronutrients (production, availability, demand, consumption and deficiency [MND]) and macronutrients (production, availability, demand, consumption and hunger) (???).
Example of a Bayesian Network for household nutrition impacts of a policy strategy (Vision 2040) in Uganda. Probabilities shown for outcome variables and variables with the highest value of information (boxes with bars). Green bars show the probabilities for node states under the decision not to implement the decision, and blue bars show probabilities for node states under the scenario that the decision is implemented.
Following the steps shown in the causal relationships identified in were programmed into a BN model. Following the steps putlined in the CPTs were then filled with calibrated expert estimates for each node and edge. This was written into a programmable BN model developed in AgenaRisk (Fenton & Neil, 2016), one of many commercial software options for BN model development. The model was published on the Harvard DataVerse along with the a detailed explanation of the data (probabilities and formulas) behind the model (see Luedeling, 2017).
Value of Information analysis
As we said in it is important for the analyst to gather all the variables that the expert group agrees are logically important to describe the impact pathway of the decision and include them in the model. This should happen for several reasons, the most important of which is that the final development outcomes are generally strongly dependent on variables with high uncertainty. These intangible and uncertain variables often include important factors such as behavioral, institutional and policy factors, and are likely to be brought up by the experts as part of their discussions of impact pathways. Part of the value of the Decision Analysis approach is that it directs attention and performance monitoring to such factors, identified by their high information value.
When a model has been programmed in the approproate software and no clearly preferable option emerges from teh model results, Value of Information (VoI) analysis is a tool that can be used to reveal what further information is needed to narrow uncertainty and clarify a decision. It is expressed as the amount that a rational decision-maker would be willing to pay for that knowledge before making a decision. VoI is a central concept of Decision Analysis, it is used to guide decisions on the level of complexity to be considered in a decision model and the need for further measurement to clarify decision alternatives. VoI can be estimated by analyzing the uncertainties in all the variables that have a bearing on a decision. According to Abbas & Howard (2015) VoI could also be called the value of clairvoyance.
VoI analysis can be used to determine whether additional information on certain input variables in the BN model could increase confidence about the emerging decision recommendation. The examples of Hubbard (2014) show that when value of information analysis is used to prioritize measurements, it is often found that only a few variables may be relevant, and data collection should focus on those that narrow choices the most. Other examples, such as those of Constantinou et al. (2016) and Whitney et al. (2017), illustrate how the results of VoI analysis can be used for prioritizing knowledge gaps that should most urgently be narrowed in order to improve certainty about a decision. Constantinou et al. (2016) also show that VoI can help decision makers learn if a decision outcome is dependent on, or independent of any risk factors, thereby helping to focus any necessary follow-up research resources. More follow-up measurements and disaggregation of any identified variables can help inform the design and prioritization of future research and provide guidance about the best pathways for implementing the current decision.
Expected Value of Perfect Information
The Expected Value of Perfect Information (EVPI) is one useful VoI tool. It is the difference between the expected value of a decision made with perfect information and the expected value of the decision with current imperfect information (Hubbard, 2014). The many examples presented by Hubbard (2014) and those of Felli & Hazen (2003) show how EVPI can help decision makers to consider both the probability of changes due to a decision, and the resulting difference in payoff. Constantinou et al. (2016) show how EVPI can be calculated for BN models to identify a selected subset of important model variables. To achieve this, utility nodes are used to assign monetary value to model outputs (example shown in ).
The Expected Monetary Value (EMV) is a key part of the EVPI calculation. It is the weighted average of the payoffs for a decision alternative, where weights are the probabilities of the different states of nature (). EVPI is the maximum amount that one should be willing to pay for additional information about the decision. \(EVPI = EV with PI – max(EMV)\), i.e. it is the expected value for the decision (payoff) if perfect information is available about the states of nature, minus the expected value for the decision if perfect information is not available.
A simple hypothetical example of a Value of Information (VoI) calculation, based on the Ugandan nutrition example, is given in . This model is shown in and available on the Harvard DataVerse (Luedeling, 2017). The table is populated with estimated values for different states of the Diversity of household diets based on the implementation of Vision 2040.
shows the calculation of EVPI in a BN. The main part of the table is populated with a ‘utility value’ for diverse diets under each of the ‘states of nature’, e.g. the upper right value of -4 represents the utility value of low household dietary diversity in the scenario where the decision to implement Vision 2040 is not taken. The likelihood of each of the states of Diversity of household diets is shown in the row labeled with Probability.
In EMV is calculated for each state of the Vision 2040 decision by adding the utility values after multiplying them by the probability for each state of Diversity of household diets. The maximum EMV is the highest of these two (27.7). Expected value with perfect information (EV with PI) is calculated for each column by selecting the highest value for each state of Diversity of household diets (29.7). EVPI is calculated using the resulting values \(EVPI = EV with PI – max(EMV)\).
Variables that have large uncertainty and a large potential impact on outcomes will have high information values. Information values also point decision makers to places where they could adjust the intervention design to reduce risks and improve outcomes, and also where to increase model complexity. With the new information, the model can be run again and the process is repeated until decision makers feel confident that they can make a well-informed decision .
Value of information analysis provides an efficient, iterative approach to information collection, as measurements are only made as far as needed and lowest cost options are tried first. As a first step … decomposing model variables into sub-variables that are easier to estimate in an attempt to narrow uncertainty. If there is still residual information value, then the analysts may try further literature review or consulting with more experts to further narrow the uncertainty.
In this approach analysts only need to design physical measurements or surveys if the other steps are insufficient. Even in teh case that more data needs to be gathered, small sample sizes may be sufficient to reduce uncertainty and reveal a clear decision. Value of information analysis also tells decision makers how much they should consider spending on these measurements. From applying probabilistic decision modeling on over 80 diverse problems, Hubbard (2014) observed that only a few variables typically had high information value in any given decision, and interestingly they were rarely variables receiving current measurement effort.
The decisionSupport package
The decisionSupport package (Luedeling & Goehring, 2017) in the R programming language (Team, 2017) performs several useful functions for the development of BNs. One particularly useful tool is the likelihood method for estimating CPTs. The the likelihood method works by defining the likelihood that states of several qualitative variables lead to states of another qualitative one.
gives an example of a tool for gathering input from experts about the translating expert knowledge into a Conditional Probability table (CPT) for use in a Bayesian Network (adapted from Whitney et al., in preparation).
For application in developing CPTs there are three useful functions within the decisionSupport package:
The make_CPT function creates Conditional Probability Tables for Bayesian Network nodes from parameters that (for complex nodes) can be more easily elicited from experts than the full table. The function uses the Likelihood method, as described by Hansson & Sjökvist (2013). Tables are created from the relative weights of all parents, rankings for all parents, a parameter (b) for the sensitivity of the child node and a prior distribution (for the child node).
The sample_CPT function randomly chooses a state of a categorical variable, based on a Conditional Probability Table (CPT; a component of Bayesian Network models) that expresses the probability of each possible state in relation to the states of other categorical variables. Given information on the state of all parent variables, the function uses the appropriate probability distribution to draw a random sample for the state of the variable of interest.
The sample_simple_CPT function creates Conditional Probability Tables for Bayesian Network nodes from parameters that (for complex nodes) can be more easily elicited from experts than the full table. The function uses the Likelihood method. The function combines the make_CPT and sample_CPT functions, but only offers limited flexibility.
All the above functions are described in detail in the decisionSupport package. Here we will demonstrate the use of the make_CPT function using data gathered with an input sheet shown in . As we described in {model-quantification} this sheet helps analysts and experts to represent qualitative nodes as abstractions of the underlying continuous quantities. shows an input sheet that has been filled by experts for building a CPT in the BN shown in . In this model the variable Food Preferences was considered important by experts, but was difficult to quantify. Therefore it is represented as having the states ‘very low’, ‘low’, ‘medium’, ‘high’, and ‘very high’. Through the use of the input sheet and the make_CPT function, the CPT for this variable can be populated with the probabilities for these different states given the influence and states of the parents Location and Nutrition awareness.
Example of a filled input sheet tool for translating expert knowledge on Food Preferences given the states of the parents Location and Nutrition awareness into a Conditional Probability table (CPT) for use in a Bayesian Network
The values gathered from experts are shown in . These can be translated into a CPT through the use of the make_CPT function in R using the R code shown below. Entering this code into R yields two outputs. The first $CPT contains conditional probabilities with the states of Food Preferences as rows. The second output from the make_CPT function is a descriptor for the columns in the CPT called $column_legend. These columns are based on the different combined states of the parents Location and Nutrition awareness.
R code with resulting CPT and legend tables
library(decisionSupport)
make_CPT(parent_effects=list(c(-1,-0.5,0, 0.5, 1), c(1,0.6,-1,-1,-1,-0.6)),
parent_names=c("Nutrition awareness","Location"),
parent_weights=c(1,1),b=2,child_prior=c(0.2,0.3,0.3, 0.1, 0.1),
child_states=c("Very low","Low","Medium", "High", "Very high"),
parent_states=list(c("Very low","Low","Medium", "High", "Very high"),
c("Same rural area","Different rural area",
"Forest, wetland etc.","Small urban area",
"Large urban area","Other countries")))## $CPT
## col_1 col_2 col_3 col_4 col_5 col_6
## Very low 0.2 0.2463258 0.45070423 0.45070423 0.45070423 0.39912497
## Low 0.3 0.3216585 0.33802817 0.33802817 0.33802817 0.34385565
## Medium 0.3 0.2800200 0.16901408 0.16901408 0.16901408 0.19749321
## High 0.1 0.0812572 0.02816901 0.02816901 0.02816901 0.03781002
## Very high 0.1 0.0707385 0.01408451 0.01408451 0.01408451 0.02171615
## col_7 col_8 col_9 col_10 col_11 col_12
## Very low 0.1482354 0.1890469 0.38612313 0.38612313 0.38612313 0.33417503
## Low 0.2644239 0.2935705 0.34438528 0.34438528 0.34438528 0.34237240
## Medium 0.3144547 0.3039232 0.20477271 0.20477271 0.20477271 0.23384724
## High 0.1246506 0.1048804 0.04058619 0.04058619 0.04058619 0.05324079
## Very high 0.1482354 0.1085790 0.02413270 0.02413270 0.02413270 0.03636453
## col_13 col_14 col_15 col_16 col_17 col_18
## Very low 0.1048709 0.1388499 0.32129166 0.32129166 0.32129166 0.27073927
## Low 0.2224648 0.2564164 0.34078126 0.34078126 0.34078126 0.32986293
## Medium 0.3146127 0.3156856 0.24096874 0.24096874 0.24096874 0.26793196
## High 0.1483098 0.1295515 0.05679688 0.05679688 0.05679688 0.07254279
## Very high 0.2097418 0.1594966 0.04016146 0.04016146 0.04016146 0.05892306
## col_19 col_20 col_21 col_22 col_23
## Very low 0.07070264 0.09729506 0.25844424 0.25844424 0.25844424
## Low 0.17836078 0.21367246 0.32598725 0.32598725 0.32598725
## Medium 0.29996588 0.31283480 0.27412151 0.27412151 0.27412151
## High 0.16816016 0.15267232 0.07683593 0.07683593 0.07683593
## Very high 0.28281055 0.22352536 0.06461106 0.06461106 0.06461106
## col_24 col_25 col_26 col_27 col_28 col_29 col_30
## Very low 0.21122204 0.04545455 0.06496624 0.2 0.2 0.2 0.1579600
## Low 0.30604056 0.13636364 0.16966918 0.3 0.3 0.3 0.2721725
## Medium 0.29561569 0.27272727 0.29541121 0.3 0.3 0.3 0.3126441
## High 0.09518198 0.18181818 0.17144693 0.1 0.1 0.1 0.1197113
## Very high 0.09193973 0.36363636 0.29850644 0.1 0.1 0.1 0.1375121
##
## $column_legend
## col_1 col_2
## Nutrition awareness "Very low" "Very low"
## Location "Same rural area" "Different rural area"
## col_3 col_4
## Nutrition awareness "Very low" "Very low"
## Location "Forest, wetland etc." "Small urban area"
## col_5 col_6 col_7
## Nutrition awareness "Very low" "Very low" "Low"
## Location "Large urban area" "Other countries" "Same rural area"
## col_8 col_9
## Nutrition awareness "Low" "Low"
## Location "Different rural area" "Forest, wetland etc."
## col_10 col_11
## Nutrition awareness "Low" "Low"
## Location "Small urban area" "Large urban area"
## col_12 col_13
## Nutrition awareness "Low" "Medium"
## Location "Other countries" "Same rural area"
## col_14 col_15
## Nutrition awareness "Medium" "Medium"
## Location "Different rural area" "Forest, wetland etc."
## col_16 col_17
## Nutrition awareness "Medium" "Medium"
## Location "Small urban area" "Large urban area"
## col_18 col_19
## Nutrition awareness "Medium" "High"
## Location "Other countries" "Same rural area"
## col_20 col_21
## Nutrition awareness "High" "High"
## Location "Different rural area" "Forest, wetland etc."
## col_22 col_23
## Nutrition awareness "High" "High"
## Location "Small urban area" "Large urban area"
## col_24 col_25
## Nutrition awareness "High" "Very high"
## Location "Other countries" "Same rural area"
## col_26 col_27
## Nutrition awareness "Very high" "Very high"
## Location "Different rural area" "Forest, wetland etc."
## col_28 col_29
## Nutrition awareness "Very high" "Very high"
## Location "Small urban area" "Large urban area"
## col_30
## Nutrition awareness "Very high"
## Location "Other countries"To help with interpritation of the R ouput we have included the full CPT for Food Preferences in together with the with input data for prior probabilities for all nodes, strength of response (b), and relative influence of the parent nodes (see Luedeling, 2017).
The resulting probabilities for Food Preferences under the scenario that the Ugandan Government’s Vision 2040 is implemented are shown in and in .
The AgenaRisk package
There are several commercial software options for modeling Bayesian Networks. These packages offer a visual interaction with the progammmable BN model. In the software package AgenaRisk, graphical models are called ‘Risk Maps’.
Interface of AgenaRisk software with example model of probabilities for different states of Food Preferences under the Ugandan Government’s decision to implement Vision 2040.
Creating a Bayesian Network in AgenaRisk
shows a screenshot of the main interface from the AgenaRisk package (Fenton & Neil, 2016). The top row of icons are for the overall model, the second row of icons are for individual nodes. The large white space to the right is where the graphical model can be drawn and the column of icons to the left offer the tools used for drawing the graphical model.
The lower three icons in the left vertical tool bar can be used for making nodes and edges. Nodes can be created that are based on CPTs (the top circle) or can be based on calculations (bottom circle, marked with \(\Sigma\)). The nodes can have a number of differet states, the symbols in the upper right corner of each of these nodes shows the type of node being used (B=boolean, R=ranked, \(\Sigma\)=arithmetic). The arrow icon from the left icon bar can be selected to connect nodes. The arrow represents causal relationships and connects parents to child nodes, thereby changing the underlying probability tables.
Each node within the graphical model includes an underlying, probability table, referred to as a Node Probability Table (NPT) in AgenaRisk (what is commonly called CPT, in the literature and in this text). The window for filling in the node values can be accessed by selecting the node and clicking on the \(\Sigma\) symbol in the second row of icons at the top of the interface shown in . The CPT can be populated with probability values manually, or by copying and pasting the CPT results from the make_CPT function. The first table of outputs from the decisionSupport tool .
For arithmetic nodes the values can also be entered manually. , for example, shows the node probability table for the arithmetic node labeled ‘Utility’ in . To enter the values for this node we select the Partitioned Expression option in NPT editing Mode. The utility values are entered manually by double clicking on each cell within the table shown in . The values shown are the same as those given in .
Example of a utility node as an partitioned expression filled with arthmetic values in the AgenaRisk software.
Value of Information calculation in AgenaRisk
As is shown in , the causal relationship between the decision node and the uncertainty node is expressed in the utility node. In order to identify the value of information we can calculate the EVPI of this decision, as per .
Example use of the Value of Information Analysis window in the AgenaRisk software.
Selecting ‘Run’ on this window will create a document titled ‘VoIReport.html’ showing the results and means of an EVPI calculation. This has the same results and a similar format to .
More details and examples of the functionality of the AgenaRisk software can be found in the user manual and in the software package (Fenton & Neil, 2016).
Conclusions
In this manual we have attempted to outline some tools for the adaptation and application of Bayesian Networks to agriculture for nutrition development contexts. Decision Analysis is an important paradigm for development research. The approach of using BN models promises to be an effective strategy for dealing with complex systems, multiple impact pathways, uncertain and incomplete information, and other current constraints to meaningful quantitative impact projections.
The many factors that determine the nutritional impact of agricultural interventions are interconnected in various ways, there are often clear causal connections between them. Most traditional research approaches, such as regression analysis, controlled trials etc. fare poorly in complex and multi-factorial situations, and results from modeling exercises that adopt deterministic perspectives on complex systems are often not credible because they rely on a host of simplifying assumptions. However, rational prioritization among ‘agriculture for nutrition’ actions requires an evaluation approach that can accommodate the complex relationships that credibly translate agricultural activities into nutritional outcomes.
We see the potential for great benefits to arise from an approach such as BNs within the Decision Analysis paradigm. These approaches can integrate existing data with expert opinion and other sources of information. This can be a considerable improvement in development research, especially for considering causal relationships in potential intervention impacts. We hope that this manual will contribute to the wider application of these tools to agriculture for nutrition decisions.
More information and resources are available through the referenced materials, in the the decisionSupport package for R (Luedeling & Goehring, 2017) and the AgenaRisk software (Fenton & Neil, 2016), as well as on the the Harvard DataVerse (Luedeling, 2017).
References
Abbas, A. E., & Howard, R. A. 2015. Foundations of decision analysis: 832. NY, NY: Prentice Hall.
Bolger, F., & Rowe, G. 2015. The aggregation of expert judgment: Do good things come to those who weight. Risk Analysis, 35(1): 5–11.
Bolger, F., & Wright, G. 2017. Use of expert knowledge to anticipate the future: Issues, analysis and directions. International Journal of Forecasting, 33(1): 230–243.
Cain, J. 2001. Planning improvements in natural resource management. guidelines for using bayesian networks to support the planning and management of development programmes in the water sector and beyond: 136. Crowmarsh Giford, Wallingford, Oxon, UK: Centre for Ecology; Hydrology.
Clemen, R. T., & Winkler, R. L. 1999. Combining probability distributions from experts in risk analysis. Risk Analysis, 19(2): 187–203.
Constantinou, A., Yet, B., Fenton, N., Neil, M., & Marsh, W. 2016. Value of information analysis for interventional and counterfactual bayesian networks in forensic medical sciences. Artificial Intelligence in Medicine, 66: 41–52.
DellaPenna, D. 1999. Nutritional genomics: Manipulating plant micronutrients to improve human health. Science, 285(5426): 375–379.
Felli, J. C., & Hazen, G. B. 2003. Sensitivity analysis and the expected value of perfect information. Medical Decision Making, 23(1): 97.
Fenton, N., & Neil, M. 2012. Risk assessment and decision analysis with bayesian networks: 503. Boca Raton, Florida: CRC Press.
Fenton, N., & Neil, M. 2016. AgenaRisk professional version 7.0. Revision 3451 VOI.
Fenton, N., Neil, M., & Caballero, J. 2007. Using ranked nodes to model qualitative judgments in bayesian networks. IEEE Transactions on Knowledge and Data Engineering, TKDE-0095-0206.R2: 1–11.
Hansson, F., & Sjökvist, S. 2013. Modelling expert judgement into a bayesian belief network. PhD thesis, Lund University, Lund, Sweden.
Henderson, J., & Burn, R. 2004. Uptake pathways: The potential of bayesian belief networks to assist the management, monitoring and evaluation of development-orientated research. Agr. Sys., 79(1): 3–15.
Hoddinott, J., & Yohannes, Y. 2002. Dietary diversity as a food security indicator, vol. Discussion Paper 136: 1–2. Washington, DC: International Food Policy Research Institute (IFPRI).
Hubbard, D. W. 2014. How to measure anything: Finding the value of intangibles in business, vol. Second Edition: 301. Hoboken, New Jersey: John Wiley & Sons.
Iqbal, J., & MacLean, D. A. 2010. Prediction of balsam fir sawfly defoliation using a bayesian network model. Can. J. for. Res., 40(12): 2322–2332.
Kemp-Benedict, E., Bharwani, S., Rosa, E. de la, Krittasudthacheewa, C., & Matin, N. 2009. Assessing water-related poverty using the sustainable livelihoods framework. Stockholm: Stockholm Environment Institute.
Korb, K. B., & Nicholson, A. E. 2004. Bayesian artificial intelligence.: 365. London: Chapman & Hall/CRC Press UK.
Kuhnert, P., Martin, T., & Griffiths, S. 2010. A guide to eliciting and using expert knowledge in bayesian ecological models. Ecology Letters, 13(7): 900–914.
Leroy, J. L., Ruel, M., & Verhofstadt, E. 2009. The impact of conditional cash transfer programmes on child nutrition: A review of evidence using a programme theory framework. Journal of Development Effectiveness, 1(2): 103–129.
Lewis, F. I. 2012. Bayesian networks as a tool for epidemiological systems analysis: 610–617. Zurich: Zurich Open Repository; Archive, University of Zurich.
Lewis, F., & McCormick, B. 2012. Revealing the complexity of health determinants in resource-poor settings. American Journal of Epidemiology, 176(11): 1051–1059.
Luedeling, E. 2017. Probabilistic causal models for nutrition outcomes of agricultural actions - uganda model. Harvard Dataverse, World Agroforestry Centre - ICRAF Dataverse, ICRAF Decision Analysis Dataverse.
Luedeling, E., & Goehring, L. 2017. DecisionSupport – quantitative support of decision making under uncertainty. contributed package to the r programming language. version 1.103.2.
Luedeling, E., & Shepherd, K. 2016. Decision-focused agricultural research. Solutions, 7(5): 46–54.
Luedeling, E., Oord, A. L., Kiteme, B., Ogalleh, S., Malesu, M., et al. 2015. Fresh groundwater for wajir - ex-ante assessment of uncertain benefits for multiple stakeholders in a water supply project in northern kenya. Frontiers in Environmental Science, 3, article 16: 1–18.
Marcot, B. G., Steventon, J. D., Sutherland, G. D., & McCann, R. K. 2006. Guidelines for developing and updating bayesian belief networks applied to ecological modeling and conservation. Can. J. for. Res., 36(12): 3063–3074.
Nestel, P., Bouis, H. E., Meenakshi, J., & Pfeiffer, W. 2006. Biofortification of staple food crops. The Journal of Nutrition, 136(4): 1064–1067.
Nielsen, T. D., & Jensen, F. V. 2009. Bayesian networks and decision graphs: XVI, 448. New York: Springer Science & Business Media.
Olney, D. K., Talukder, A., Iannotti, L. L., Ruel, M. T., & Quinn, V. 2009. Assessing impact and impact pathways of a homestead food production program on household and child nutrition in cambodia. Food and Nutrition Bulletin, 30(4): 355–369.
O’Hagan, A., Buck, C. E., Daneshkhah, A., Eiser, J. R., Garthwaite, P. H., et al. 2006. Uncertain judgements: Eliciting experts’ probabilities. West Sussex: John Wiley & Sons.
Papakosta, P., Xanthopoulos, G., & Straub, D. 2017. Probabilistic prediction of wildfire economic losses to housing in cyprus using bayesian network analysis. International Journal of Wildland Fire, 26(1): 10–23.
Ruel, M., Alderman, H., & Maternal, C. N. S. G. 2013. Nutrition-sensitive interventions and programmes: How can they help to accelerate progress in improving maternal and child nutrition. Lancet, 382(9891): 536–551.
Shepherd, K., Hubbard, D., Fenton, N., Claxton, K., Luedeling, E., et al. 2015. Development goals should enable decision-making. Nature, 523(7559): 152–154.
Shepherd, K., Luedeling, E., & Whitney, C. W. in preparation. A decision analysis framework for development planning and performance measurement: Application to land restoration investments.
Team, R. C. 2017. R: A language and environment for statistical computing [r version 3.4.1 (2017-06-30) “single candle”], 3.4.1.
Waage, J., Hawkes, C., & Turner, R. 2012. Current and planned research on agriculture for improved nutrition: A mapping and a gap analysis. a report for dfid: 48. London: Leverhulme Centre for Integrative Research on Agriculture; Health (LCIRAH).
Whitney, C. W., Lanzanova, D., Muchiri, C., Shepherd, K. D., Rosenstock, T. S., et al. in preparation. Probabilistic decision tools for determining impacts of agricultural development policy on household nutrition.
Whitney, C. W., Tabuti, J. R., Hensel, O., Yeh, C.-h., Gebauer, J., et al. 2017. Homegardens and the future of food and nutrition security in southwest uganda. Agricultural Systems, 154: 133–144.
Yet, B., Constantinou, A., Fenton, N., Neil, M., Luedeling, E., et al. 2016. A bayesian network framework for project cost, benefit and risk analysis with an agricultural development case study. Expert Systems with Applications, 60: 141–155.